Papers by Hyung Won Chung
Learning Compact Metrics for MT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. |
| Approach: | They propose to use multilingual model capacity to improve model performance by transferring knowledge from one teacher to multiple students trained on related languages. |
| Outcome: | The proposed model yields 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters. |
A Simple and Effective Positional Encoding for Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that relative position encodings provide better performance than absolute position coding. |
| Approach: | They propose a mechanism to encode position and segment information into Transformer models using relative position encodings. |
| Outcome: | The proposed method achieves faster training and inference time while achieving competitive performance on GLUE, XTREME and WMT benchmarks. |
Improving Multilingual Models with Language-Clustered Vocabularies (2020.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art multilingual models depend on vocabularies that cover all languages . but the methods for generating those vocalaries are not ideal for massively multilingual applications. |
| Approach: | They propose a procedure for multilingual vocabulary generation that combines separately trained vocabularies of several automatically derived language clusters. |
| Outcome: | The proposed procedure shows improvements across languages on multilingual benchmark tasks . the proposed procedure reduces out-of-vocabulary rate by a factor of 8 . |
Do Transformer Modifications Transfer Across Implementations and Applications? (2021.emnlp-main)
Copied to clipboard
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
| Challenge: | Currently, the Transformer is the de facto architecture of choice for processing sequential data. |
| Approach: | They evaluate the Transformer architecture and its modifications in a shared experimental setting . they conjecture that performance improvements may strongly depend on implementation details . |
| Outcome: | The proposed improvements do not significantly improve performance, the authors find . the proposed improvements are either developed in the same codebase or are minor changes . |
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2023.findings-acl)
Copied to clipboard
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, Jason Wei
| Challenge: | Language models have already made good progress on this benchmark, with the best model outperforming average reported human-rater results on 65% of the BIG-Bench tasks. |
| Approach: | They propose to use chain-of-thought prompting to challenge language models on 23 challenging BIG-Bench tasks which they call BIG-Bench Hard. |
| Outcome: | The proposed language models outperform the average human-rater on 65% of the BIG-Bench tasks. |